Sound processing registering a word in a dictionary

ABSTRACT

The present invention relates to a voice recognition apparatus capable of easily registering a word which has not been registered. The registering of an unregistered word into a dictionary can be easily performed without causing a significant increase in the size of the dictionary. The clustering unit detects a cluster (detected cluster) to which a new unregistered word is to be added as a new member, from existing clusters obtained by clustering unregistered words. The unregistered word is added as a new member to the detected cluster, and the cluster is divided depending on the members of the cluster such that unregistered words which are acoustically similar to each other belong to the same cluster. The maintenance unit updates the word dictionary on the basis of the result of the clustering. The present invention may be applied to a robot including a voice recognition apparatus.

This application is a 371 of PCT/JPT/02/03248 Apr. 1, 2002

TECHNICAL FIELD

The present invention relates to a voice recognition apparatus, and moreparticularly, to a voice recognition apparatus capable of easilyupdating a dictionary in which a word or a phrase to be recognized isregistered.

BACKGROUND ART

In conventional voice recognition apparatuses, recognition of a voiceuttered by a user is performed by referring to a dictionary in whichwords to be recognized are registered.

Therefore, in the voice recognition apparatus, only words which areregistered in the dictionary (hereinafter, such words will be referredto simply as registered words) can be recognized, and words which arenot registered in the dictionary cannot be recognized. Herein, wordswhich are not registered in the dictionary are referred to asunregistered words. In the conventional voice recognition apparatus, ifan utterance made by a user includes an unregistered word, theunregistered word is recognized as one of words (registered words)registered in the dictionary, and thus the result of recognition of theunregistered word becomes wrong. If an unregistered word is recognizedincorrectly, the incorrect recognition can influence recognition of aword prior to or subsequent to the unregistered word, that is, can causesuch a word to be recognized incorrectly.

Therefore, it is required to properly deal with unregistered words so asto avoid the above problem. To this end, various techniques have beenproposed.

For example, Japanese Unexamined Patent Application Publication No.9-81181 discloses a voice recognition apparatus in which a garbage modelfor detecting an unregistered word and an HMM (Hidden Markov Model)associated with phonemes such as vowels are simultaneously used so as tolimit phoneme sequences associated with the unregistered word therebymaking it possible to detect the unregistered word without needingcomplicated calculations.

As another example, Japanese Patent Application No. 11-245461 disclosesan information processing apparatus in which when a word set includingan unregistered word is given, the similarity between the unregisteredword which is not included in a database and a word included in thedatabase is calculated on the basis of the concepts of words, and asequence of properly arranged words is produced and output.

As still another example, “Dictionary Learning: Performance ThroughConsistency” (Tilo Sloboda, Proceedings of ICASSP 95, vol. 1, pp.453–456, 1995) discloses a technique in which phoneme sequencescorresponding to voice periods of words are detected and phonemesequences which are acoustically similar to each other are deleted usinga confusion matrix thereby effectively constructing a dictionaryincluding variants.

As still another example, “Estimation of Transcription of Unknown Wordfrom Speech Samples in Word Recognition” (Katsunobu Ito, et at., TheTransactions of the Institute of Electronics, Information, andCommunication Engineers, Vol. J83-D-II, No. 11, pp. 2152–2159, November,2000) discloses a technique of improving estimation accuracy of aphoneme sequence when the phoneme sequence is estimated from a pluralityof speech samples and an unknown (unregistered) word is registered in adictionary.

One typical method for dealing with an unregistered word is to, if anunregistered word is detected in an input voice, register theunregistered word into a dictionary and treat it as an registered wordthereafter.

In order to register an unregistered word into a dictionary, it isrequired to first detect a voice period of that unregistered word andthen recognize the phoneme sequence of the voice in the voice period.The recognition of the phoneme sequence of a voice can be accomplished,for example, by a method known as a phoneme typewriter. In the phonemetypewriter, a phoneme sequence corresponding to an input voice isbasically output using a garbage model which accepts any phonemicchange.

When an unregistered word is registered into a dictionary, it isrequired to cluster the phoneme sequence of the unregistered word. Thatis, in the dictionary, the phoneme sequence of each word is registeredin the form of a cluster corresponding to the word, and thus, toregister an unregistered word into the dictionary, it is required tocluster the phoneme sequence of the unregistered word.

One method of clustering the phoneme sequence of an unregistered word isto input, by a user, an entry (for example, a pronunciation of theunregistered word) indicating the unregistered word and then cluster thephoneme sequence of the unregistered word into a cluster indicated bythe that entry. However, in this method, the user has to do atroublesome task to input the entry.

Another method is to produce a new cluster each time an unregisteredword is detected such that the phoneme sequence of the unregistered wordis clustered into the newly produced cluster. However, in this method,an entry corresponding to the new cluster is registered into adictionary each time an unregistered word is detected, and thus the sizeof the dictionary increases as unregistered words are registered. As aresult, a greater time and a greater amount of process are necessary invoice recognition performed thereafter.

DISCLOSURE OF INVENTION

In view of the above, it is an object of the present invention toprovide a technique of easily registering an unregistered word into adictionary without causing a significant increase in the size of thedictionary.

The present invention provides a voice recognition apparatus comprisingcluster detection means for detecting, from existing clusters obtainedby clustering voices, a cluster to which an input voice is to be addedas a new member; cluster division means for employing the input voice asthe new member of the cluster detected by the cluster detection meansand dividing the cluster depending on members of the cluster; and updatemeans for updating the dictionary on the basis of a result of divisionperformed by the cluster division means.

The present invention provides a voice recognition method comprising thesteps of detecting, from existing clusters obtained by clusteringvoices, a cluster to which the input voice is to be added as a newmember; employing the input voice as the new member of the clusterdetected in the cluster detection step and dividing the clusterdepending on members of the cluster; and updating the dictionary on thebasis of a result of division performed in the cluster division step.

The present invention provides a program comprising the steps ofdetecting, from existing clusters obtained by clustering voices, acluster to which the input voice is to be added as a new member;employing the input voice as the new member of the cluster detected inthe cluster detection step and dividing the cluster depending on membersof the cluster; and updating the dictionary on the basis of a result ofdivision performed in the cluster division step.

The present invention provides a storage medium including a program,stored therein, comprising the steps of detecting, from existingclusters obtained by clustering voices, a cluster to which the inputvoice is to be added as a new member; employing the input voice as thenew member of the cluster detected in the cluster detection step anddividing the cluster depending on members of the cluster; and updatingthe dictionary on the basis of a result of division performed in thecluster division step.

In the present invention, from existing clusters obtained by clusteringvoices, a cluster to which an input voice is to be added as a new memberis detected. The input voice is added as a new member to the detectedcluster and the cluster is divided depending on the members of thecluster. In accordance with a result of the division, the dictionary isupdated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing an example of an outward structureof a robot according to an embodiment of the present invention.

FIG. 2 is a block diagram showing an example of an internal structure ofthe robot.

FIG. 3 is a block diagram showing an example of a functional structureof a controller of the robot shown in FIG. 1.

FIG. 4 is a block diagram showing an example of a construction of avoice recognition apparatus according to an embodiment of the presentinvention, wherein the voice recognition apparatus is used as a voicerecognition unit of the robot shown in FIG. 1.

FIG. 5 is a diagram showing a word dictionary.

FIG. 6 is a diagram showing grammatical rules.

FIG. 7 is a diagram showing contents stored in a feature vector bufferof the voice recognition unit shown in FIG. 4.

FIG. 8 is a diagram showing a score sheet.

FIG. 9 is a flow chart showing a voice recognition process performed bythe voice recognition unit shown in FIG. 4.

FIG. 10 is a flow chart showing the details of unregistered wordprocessing shown in FIG. 9.

FIG. 11 is a flow chart showing the details of a cluster divisionprocess shown in FIG. 9.

FIG. 12 is a diagram showing a result of a simulation.

FIG. 13 is a diagram showing an example of a hardware configuration of avoice recognition apparatus according to a second embodiment of thepresent invention.

FIG. 14 is a diagram showing an example of a software configuration ofthe voice recognition apparatus shown in FIG. 13.

FIG. 15 is a diagram showing contents stored in a feature vector bufferof the voice recognition unit shown in FIG. 14.

FIG. 16 is a flow chart showing a voice recognition process performed bythe voice recognition apparatus shown in FIG. 14.

FIG. 17 is a flow chart showing the details of an unregistered worddeletion process shown in FIG. 16.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1 shows an example of an outward structure of a robot according toan embodiment of the present invention, and FIG. 2 shows an example ofan electric configuration thereof.

In the present embodiment, the robot is constructed into the form of ananimal having four legs, such as a dog, wherein leg units 3A, 3B, 3C,and 3D are attached, at respective four corners, to a body unit 2, and ahead unit 4 and a tail unit 5 are attached, at front and bock ends, tothe body unit 2.

The tail unit 5 extends from a base 5B disposed on the upper surface ofthe body unit 2 such that the tail unit 5 can bend or shake with twodegree of freedom.

Inside the body unit 2, there are disposed a controller 10 for generallycontrolling the robot, a battery 11 serving as a power source of therobot, and an internal sensor unit 14 including a battery sensor 12 anda heat sensor 13.

On the head unit 4, there are disposed, at properly selected position, amicrophone 15 serving as an ear, a CCD (Charge Coupled Device) camera 16serving as an eye, a touch sensor 17 serving as a sense-of-touch sensor,and a speaker 18 serving as a mouth. A lower jaw unit 4A serving as alower jaw of the mouth is attached to the head unit 4 such that thelower jaw unit 4A can move with one degree of freedom. The mouth of therobot can be opened and closed by moving the lower jaw unit 4A.

As shown in FIG. 2, actuators 3AA₁ to 3AA_(K), 3BA₁ to 3BA_(K), 3CA₁ to3CA_(K), 3DA₁ to 3DA_(K), 4A₁ to 4A_(L), 5A₁, and 5A₂ are respectivelydisposed in joints for joining parts of the leg units 3A to 3D, jointsfor joining the leg units 3A to 3D with the body unit 2, a joint forjoining the head unit 4 with the body unit 2, a joint for joining thehead unit 4 with the lower jaw unit 4A, and a joint for joining the tailunit 5 with the body unit 2.

The microphone 15 disposed on the head unit 4 collects a voice (sound)including an utterance made by a user from the environment and transmitsan obtained voice signal to the controller 10. The CCD camera 16 takesan image of the environment and transmits an obtained image signal tothe controller 10.

The touch sensor 17 is disposed on an upper part of the head unit 4. Thetouch sensor 17 detects a pressure applied by the user as a physicalaction such as “rubbing” or “tapping” and transmits a pressure signalobtained as the result of the detection to the controller 10.

The battery sensor 12 disposed in the body unit 2 detects the remainingcapacity of the battery 11 and transmits the result of the detection asa battery remaining capacity signal to the controller 10.

The heat sensor 13 detects heat in the inside of the robot and transmitsinformation indicating the detected heat, as a heat signal, to thecontroller 10.

The controller 10 detects a CPU (Central Processing Unit) 10A and amemory 10B. The controller 10 performs various processes by executing,using the CPU 10A, a control program stored in the memory 10B.

The controller 10 detects an environment state, a command given by auser, and an action of a user applied to the robot on the basis of thevoice signal, the image signal, the pressure signal, the batteryremaining capacity signal, and the heat signal supplied from themicrophone 15, the CCD camera 16, the touch sensor 17, the batterysensor 12, and the heat sensor 13, respectively.

On the basis of the parameters detected above, the controller 10 makes adecision as to how to act. In accordance with the decision, thecontroller 10 activates necessary actuators of those including actuators3AA₁ to 3AA_(K), 3BA₁ to 3BA_(K), 3CA₁ to 3CA_(K), 3DA₁ to 3DA_(K), 4A₁to 4A_(L), 5A₁, and 5A₂, so as to nod or shake the head unit 4 or openand close the lower jaw unit 4A. Depending on the situation, thecontroller 10 moves the tail unit 5 or makes the robot walk by movingthe leg units 3A to 3D.

Furthermore, as required, the controller 10 produces synthesized voicedata and supplies it to the speaker 18 thereby generating a voice, orturns on/off or blinks LEDs (Light Emitting Diode, not shown in thefigures) disposed on the eyes of the robot.

As described above, the robot autonomously acts in response to theenvironmental conditions.

FIG. 3 shows the functional structure of the controller 10 shown in FIG.2. Note that the functional structure shown in FIG. 3 is realized byexecuting, using the CPU 10A, the control program stored in the memory10B.

The controller 10 includes a sensor input processing unit 50 forrecognizing a specific external state, a model memory 51 for storingresults of recognition performed by the sensor input processing unit 50and representing a state of emotion, instinct, or growth, an actiondecision unit 52 for determining an action to be taken next inaccordance with the result of recognition performed by the sensor inputprocessing unit 50, an attitude change mechanism 53 for making the robotactually take an action in accordance with the decision made by theaction decision unit 52, a control unit 54 for controlling actuators3AA₁ to 5A₁ and 5A₂, and a voice synthesis unit 55 for producing asynthesized voice.

The sensor input processing unit 50 detects specific externalconditions, an action of a user applied to the robot, and a commandgiven by the user, on the basis of the voice signal, the image signal,and the pressure signal supplied from the microphone 15, the CCD camera16, and the touch sensor 17, respectively. Information indicating thedetected conditions is supplied as state recognition information to themodel memory 51 and the action decision unit 52.

More specifically, the sensor input processing unit 50 includes a voicerecognition unit 50A for recognizing the voice signal supplied from themicrophone 15. For example, if a given voice signal is recognized by thevoice recognition unit 50A as a command such as “walk”, “lie down”, or“follow the ball”, the recognized command is supplied as staterecognition information from the voice recognition unit 50A to the modelmemory 51 and the action decision unit 52.

The sensor input processing unit 50 also includes an image recognitionunit 50B for recognizing an image signal supplied from the CCD camera16. For example, if the image recognition unit 50B detects, via theimage recognition process, “something red and round” or a “planeextending vertical from the ground to a height greater than apredetermined value”, then the image recognition unit 50B suppliesinformation indicating the recognized state of the environment such as“there is a ball” or “there is a wall” as state recognition informationto the model memory 51 and the action decision unit 52.

The sensor input processing unit 50 further includes a pressureprocessing unit 50C for processing a detected pressure signal suppliedfrom the touch sensor 17. For example, if the pressure processing unit50C detects a pressure higher than a predetermined threshold for a shortduration, the pressure processing unit 50C recognizes that the robot hasbeen “tapped (scolded)”. In a case in which the detected pressure islower in magnitude than a predetermined threshold and long in duration,the pressure processing unit 50C recognizes that that the robot has been“rubbed (praised)”. Information indicating the result of recognition issupplied as state recognition information to the model memory 51 and theaction decision unit 52.

The model memory 51 stores and manages an emotion model, an instinctmodel, and a growth model representing the states of the robotconcerning emotion, instinct, and growth, respectively.

The emotion model represents the state (degree) of emotion concerning,for example, “happiness”, “sadness”, “angriness”, and “pleasure” usingvalues within predetermined ranges (for example, from −1.0 to 1.0),wherein the values are varied depending on the state recognitioninformation supplied from the sensor input processing unit 50 anddepending on the passage of time. The instinct model represents thestate (degree) of instinct concerning, for example, “appetite”, “desirefor sleep”, and “desire for exercise” using values within predeterminedranges, wherein the values are varied depending on the state recognitioninformation supplied from the sensor input processing unit 50 anddepending on the passage of time. The growth model represents the state(degree) of growth, such as “childhood”, “youth”, “middle age” and “oldage” using values within predetermined ranges, wherein the values arevaried depending on the state recognition information supplied from thesensor input processing unit 50 and depending on the passage of time.

The states of emotion, instinct, and growth, represented by values ofthe emotion model, the instinct model, and the growth model,respectively, are supplied as state information from the model memory 51to the action decision unit 52.

In addition to the state recognition information supplied from thesensor input processing unit 50, the model memory 51 also receives, fromthe action decision unit 52, action information indicating a current orpast action of the robot, such as “walked for a long time”, therebyallowing the model memory 51 to produce different state information forthe same state recognition information, depending on the robot's actionindicated by the action information.

More specifically, for example, when the robot greets the user, if theuser rubs the head of the robot, then action information indicating thatthe robot greeted the user and state recognition information indicatingthat the head was rubbed are supplied to the model memory 51. In thiscase, in response, the model memory 51 increases the value of theemotion model indicating the degree of happiness.

On the other hand, if the robot is rubbed on the head when the robot isdoing a job, action information indicating that the robot is doing a joband state recognition information indicating that the head was rubbedare supplied to the model memory 51. In this case, the model memory 51does not increase the value of the emotion model indicating the degreeof “happiness”.

As described above, the model memory 51 sets the values of the emotionmodel on the basis of not only the state recognition information butalso the action information indicating the current or past action of therobot.

This prevents the robot from having an unnatural change in emotion. Forexample, even if the user rubs the head of the robot with intension ofplaying a trick on the robot when the robot is doing some task, thevalue of the emotion model associated with “happiness” is not increasedunnaturally.

For the instinct model and the growth model, the model memory 51 alsoincreases or decreases the values on the basis of both the staterecognition information and the action information, as for the emotionmodel. Furthermore, when the model memory 51 increases or decreases avalue of one of the emotion model, the instinct model, and the growthmodel, the values of the other models are taken into account.

The action decision unit 52 decides an action to be taken next on thebasis of the state recognition information supplied from the sensorinput processing unit 50, the state information supplied from the modelmemory 51, and the passage of time. The content of the decided action issupplied as action command information to the attitude changing unit 53.

More specifically, the action decision unit 52 manages a finiteautomaton, which can take states corresponding to the possible actionsof the robot, as an action model which determines the action of therobot such that the state of the finite automaton serving as the actionmodel is changed depending on the state recognition information suppliedfrom the sensor input processing unit 50, the values of the model memory51 associated with the emotion model, the instinct model, and the growthmodel, and the passage of time, and the action decision unit 52 employsthe action corresponding to the changed state as the action to be takennext.

In the above process, when the action decision unit 52 detects aparticular trigger, the action decision unit 52 changes the state. Morespecifically, the action decision unit 52 changes the state, forexample, when the period of time in which the action corresponding tothe current state has been performed has reached a predetermined value,or when specific state recognition information has been received, orwhen the value of the state of the emotion, instinct, or growthindicated by the state information supplied from the model memory 51becomes lower or higher than a predetermined threshold.

Because, as described above, the action decision unit 52 changes thestate of the action model not only depending on the state recognitioninformation supplied from the sensor input processing unit 50 but alsodepending on the values of the emotion model, the instinct model, andthe growth model of the model memory 51, the state to which the currentstate is changed can be different depending on the values (stateinformation) of the emotion model, the instinct model, and the growthmodel even when the same state recognition information is input.

For example, when the state information indicates that the robot is not“angry” and is not “hungry”, if the state recognition informationindicates that “a user's hand with its palm facing up is held in frontof the face of the robot”, the action decision unit 52 produces, inresponse to the hand being held in front of the face of the robot,action command information indicating that shaking should be performedand the action decision unit 52 transmits the produced action commandinformation to the attitude changing unit 53.

On the other hand, for example, when the state information indicatesthat the robot is not “angry” but “hungry”, if the state recognitioninformation indicates that “a user's hand with its palm facing up isheld in front of the face of the robot”, the action decision unit 52produces, in response to the hand being held in front of the face of therobot, action command information indicating that the robot should lickthe palm of the hand and the action decision unit 52 transmits theproduced action command information to the attitude changing unit 53.

When the state information indicates that the robot is angry, if thestate recognition information indicates that “a user's hand with itspalm facing up is held in front of the face of the robot”, the actiondecision unit 52 produces action command information indicating that therobot should turn its face aside regardless of whether the stateinformation indicates that the robot is or is not “hungry”, and theaction decision unit 52 transmits the produced action commandinformation to the attitude changing unit 53.

In addition to above-described action command information associatedwith motions of various parts of the robot such as the head, hand, legs,etc., the action decision unit 52 also produces action commandinformation for causing the robot to utter. The action commandinformation for causing the robot to utter is supplied to the voicesynthesizing unit 55. The action command information supplied to thevoice synthesizing unit 55 includes a text or the like corresponding toa voice to be synthesized by the voice synthesizing unit 55. If thevoice synthesizing unit 55 receives the action command information fromthe action decision unit 52, the voice synthesizing unit 55 produces asynthesized voice in accordance with the text included in the actioncommand information and supplies it to the speaker 18, which in turnsoutputs the synthesized voice.

Thus, the speaker 18 outputs a voice of a cry, a voice “I am hungry” torequest the user for something, or a voice “What?” to respond to a callfrom the user. When a synthesized voice is output, the action decisionunit 52 produces action command information to open and close the lowerjaw unit 4A as required and supplies it to the attitude changing unit53. In response, the lower jaw unit 4A is opened and closed insynchronization with outputting of the synthesized voice. This can givethe user an impression that the robot is actually speaking.

In accordance with the action command information supplied from theaction decision unit 52, the attitude changing unit 53 produces attitudechange command information for changing the attitude of the robot fromthe current attitude to a next attitude and transmits it to the controlunit 54.

In accordance with the attitude change command information received fromthe attitude changing unit 53, the control unit 54 produces a controlsignal for driving the actuators 3AA₁ to 5A₁ and 5A₂ and transmits it tothe actuators 3AA₁ to 5A₁ and 5A₂. Thus, in accordance with the controlsignal, the actuators 3AA₁ to 5A₁ and 5A₂ are driven such that the robotacts autonomously.

FIG. 4 shows an example of a construction of the voice recognition unit50A shown in FIG. 3.

A voice signal output from the microphone 15 is supplied to an AD(Analog Digital) converter 21. The AD converter 21 samples and quantizesthe voice signal in the form of an along signal supplied from themicrophone 15 so as to convert it into voice data in the form of adigital signal. The resultant voice data is supplied to a featureextraction unit 22.

The feature extraction unit 22 performs, for example, MFCC (MelFrequency Cepstrum Coefficient) analysis on voice data input thereto ona frame-by-frame basis, wherein units of frames are properly selected.The MFCCs obtained via the analysis are output in the form of a featurevector (feature parameters) to a matching unit 23 and an unregisteredword period processing unit 27. Alternatively, the feature extractionunit 22 may extract, as a feature vector, linear predictioncoefficients, cepstrum coefficients, a line spectrum pair, or power foreach of particular frequency bands (output from filter banks).

Using the feature vector supplied from the feature extraction unit 22and referring, as required, to the acoustic model memory 24, thedictionary memory 25, and the grammar memory 26, the matching unit 23recognizes the voice (input voice) input to the microphone 15 on thebasis of, for example, a continuous density HMM (Hidden Markov Model).

The acoustic model memory 24 stores an acoustic model representingacoustic features of respective subwords such as phonemes and syllablesin a language of the voice to be recognized (wherein the acoustic modelmay include, in addition to the HMM, standard patterns used in the DP(Dynamic Programming) matching)). In the present embodiment, voicerecognition is performed on the basis of the continuous density HMMmethod, and thus the HMM (Hidden Markov Model) is used as the acousticmodel.

The dictionary memory 25 stores a dictionary in which words to berecognized are described in such a manner that the entry of each word isrelated to information (phonemic information) indicating thepronunciation of that word in the form of a cluster of phonemes.

FIG. 5 shows the word dictionary stored in the dictionary memory 25.

In the word dictionary, as shown in FIG. 5, entries of respective wordsare related to corresponding phoneme sequences, wherein the phonemesequences are given in the form of clusters corresponding to therespective words. In the word dictionary shown in FIG. 5, one entry (onerow in FIG. 3) corresponds to one cluster.

In the example shown in FIG. 5, each entry is represented in Romancharacters and also in Japanese characters (kanji and kana characters),and each phoneme sequence is represented in Roman characters. (In thisexample, Japanese words “

”, “

”, “

”, “

”, “

”, “

”, “

”, and “

” are pronounced “boku”, “chigau”, “doko”, “genki”, “iro”, “janai”,“kirai”, and “kudasai”, respectively, and correspond to English words“I”, “different”, “where”, “fine”, “color”, “not”, “dislike”, and“please”, respectively.) In the representations of the phonemesequences, “N” indicates a syllabic nasal corresponding to a Japanesephonetic symbol “

”. Although in the example shown in FIG. 5, one phoneme sequence isdescribed in association with each entry, a plurality of phonemesequences may be described in association with one entry.

Referring again to FIG. 4, the grammar memory 26 stores grammaticalrules indicating how those words registered in the word dictionarystored in the dictionary memory 25 are concatenated (connected).

FIG. 6 shows grammatical rules stored in the grammar memory 26. Thegrammatical rules shown in FIG. 6 are described in an EBNF (ExtendedBackus Naur Form).

In FIG. 6, one grammatical rule is described from the beginning of eachrow to a point at which “;” first appears. Each alphabetic character(character string) prefixed with “$” denotes a variable, and eachalphabetic character (character string) with no prefix of “$” representsan entry (corresponding to an entry described in Roman characters inFIG. 5) of a word. Square brackets [ ] are used to indicate that a partenclosed therein can be omitted. “|” is used to indicate that either oneof words (or variables) located before and after “|” of an entry can beselected.

More specifically, for example, in the first row of FIG. 6, thegrammatical rule “$col=[kono|sono] iro wa;” indicates that variable $colis a word string of “kono iro wa” or “sono iro wa”. (Herein, “kono” and“sono” are Japanese words corresponding to English words “this” and“that”, respectively, and “wa” is a particle in the Japanese languagehaving no corresponding English word.)

Although variables $sil and $garbage are not defined in the grammaticalrules shown in FIG. 6, variable $sil denotes a silent acoustic model(silent model), and variable $garbage denotes a garbage model whichbasically allows an arbitrary transition among phonemes.

Referring again to FIG. 4, the matching unit 23 creates an acousticmodel of a word (word model) by concatenating acoustic models stored inthe acoustic model memory 24 in accordance with the word dictionarystored in the dictionary memory 25. Furthermore, the matching unit 23concatenates some word models in accordance with grammatical rulesstores in the grammar memory 26, and the matching unit 23 recognizes thevoice input to the microphone 15 using the concatenated word models andthe feature vectors in accordance with the continuous density HMMmethod.

More specifically, the matching unit 23 detects a sequence of wordmodels having a highest score (likelihood) indicating the probabilitythat a time sequence of feature vectors output from the featureextraction unit 22 is observed, and a sequence of entries correspondingto the sequence of word models is output as a result of voicerecognition.

More specifically, the matching unit 23 determines the score for thesequence of word models by calculating the sum of occurrenceprobabilities (output probabilities) of the feature vectors of thesequence or words corresponding to the concatenated word models, and thematching unit 23 outputs, as a result of voice recognition, a sequenceof entries corresponding to a sequence of word models having a highestscore.

The result of the voice recognition on the voice input to the microphone15 is output as state recognition information to the model memory 51 andthe action decision unit 52.

In the example shown in FIG. 6, a grammatical rule using variable$garbage indicating a garbage model (hereinafter, such a grammaticalrule will be referred to as a grammatical rule including unregisteredword) “$part1=$color1 $garbage $color2;” is described in the ninth row(as counted from the top row). Thus, when this grammatical ruleincluding unregistered word is applied, the matching unit 23 detects avoice period corresponding to variable $garbage as a voice period of anunregistered word. The matching unit 23 then detects a phoneme sequenceof the unregistered word by detecting a phoneme sequence correspondingto variable $garbage described in the grammatical rule includingunregistered word, wherein variable $garbage accepts arbitrary phonemictransitions. The voice period and the phoneme sequence of theunregistered word detected as the result of the voice recognitionaccording to the grammatical rule including unregistered word aresupplied from the matching unit 23 to the unregistered word periodprocessing unit 27.

Although in the example described above, when the grammatical ruleincluding unregistered word “$part1=$color1 $garbage $color2;” isapplied, one unregistered word is detected between a phoneme sequence ofa word (word sequence) indicated by variable $color1 and registered inthe word dictionary and a phoneme sequence of a word (word sequence)indicated by variable $color2 and registered in the word dictionary, thepresent invention can also be applied to a case in which an utteranceincludes a plurality of unregistered words or to a case in which anunregistered word is not located between words (word sequences)registered in the word dictionary.

The unregistered word period processing unit 27 temporarily stores asequence of feature vectors (feature vector sequence) supplied from thefeature extraction unit 22. If the unregistered word period processingunit 27 receives the voice period and the phoneme sequence of theunregistered word from the matching unit 23, the unregistered wordperiod processing unit 27 detects a feature vector sequence of the voicein that voice period from the temporarily stored feature vectorsequence. The unregistered word period processing unit 27 assigns aunique ID (identification) to the phoneme sequence (unregistered word)supplied from the matching unit 23 and supplies it, together wit thephoneme sequence of the unregistered word and the feature vectorsequence of the voice period, to the feature vector buffer 28.

The feature vector buffer 28 temporarily stores the ID, the phonemesequence, and the feature vector sequence of the unregistered word,supplied from unregistered word period processing unit 27, such thatthey are related to each other as shown in FIG. 7.

In FIG. 7, sequential numerals starting from 1 are assigned as IDs torespective unregistered words. Therefore, for example, when the featurevector buffer 28 includes IDs, phoneme sequences, and feature vectorsequences for N unregistered words already stored therein, if thematching unit 23 detects a voice period and a phoneme sequence ofanother unregistered word, the unregistered word period processing unit27 assigns N+1 as an ID to the detected unregistered word, and the ID,the phoneme sequence, and the feature vector sequence of thatunregistered word are stored into the feature vector buffer 28 asrepresented by broken lines in FIG. 7.

Referring again to FIG. 4, the clustering unit 29 calculates the scoresfor the unregistered word newly stored in the feature vector buffer 28(hereinafter, such an unregistered word will be referred to as a newunregistered word) with respect to the other unregistered words alreadystored in the feature vector buffer 28 (hereinafter, such anunregistered word will be referred to as an already-stored unregisteredword).

That is, the clustering unit 29 regards the new unregistered word as aninput voice and the already-stored unregistered words as wordsregistered in the word dictionary, and the clustering unit 29 calculatesthe scores of the new unregistered word with respect to the respectivealready-stored unregistered words, in a similar manner as with thematching unit 23. More specifically, the clustering unit 29 detects thefeature vector sequence of the new unregistered word by referring to thefeature vector buffer 28, and concatenates acoustic models in accordancewith the phoneme sequences of the already-stored unregistered words. Theclustering unit 29 then calculates the scores indicating the likelihoodthat the feature vector sequence of the new unregistered word isobserved in the concatenated acoustic models.

As for the acoustic models used in the above process, those stored inthe acoustic model memory 24 are employed.

Similarly, the clustering unit 29 calculates the score of eachalready-stored unregistered word with respect to the new unregisteredword and updates the score sheet stored in the score sheet memory 30using the calculated scores.

The clustering unit 29 refers to the updated score sheet and detects,from existing clusters obtained by clustering unregistered words(already-stored unregistered words), a cluster to which the newunregistered word is to be added as a new member. The clustering unit 29adds the new unregistered word as a new member to the detected clusterand divides that cluster depending on the members of that cluster. Theclustering unit 29 then updates the score sheet stored in the scoresheet memory 30 on the basis of the result of the division.

The score sheet memory 30 stores the score sheet in which scores of thenew unregistered word with respect to the already-stored unregisteredwords and scores of the already-stored unregistered words associated thenew unregistered word are registered.

FIG. 8 shows the score sheet.

The score sheet includes entries in which the “ID”, “phoneme sequence”,“cluster number”, “representative member ID”, and “score” of eachregistered word are described.

For each unregistered word, the same “ID” and “phoneme sequence” asthose stored in the feature vector buffer 28 are registered by theclustering unit 29 into the score sheet. The “cluster number” is anumeral identifying a cluster including, as a member, an unregisteredword of an entry, wherein the cluster number is assigned by theclustering unit 29 and registered in the score sheet. The“representative member ID” is an ID of an unregistered word employed asa representative member of a cluster including, as a member, theunregistered word of the entry. On the basis of a representative memberID, it is possible to identify the representative member of a clusterincluding, as a member, an unregistered word. The representative memberof a cluster is determined by the clustering unit 29, and the ID of therepresentative member is registered in a representative member ID fieldin the score sheet. The “score” is a score of an unregistered word of anentry with respect to each of the other unregistered words. As describedearlier, the score is calculated by the clustering unit 29.

For example, if IDs, phoneme sequences, and feature vector sequences ofN unregistered words are currently stored in the feature vector buffer28, then the score sheet includes IDs, phoneme sequences, clusternumbers, representative member IDs, and scores of those N unregisteredwords.

When the ID, the phoneme sequence, and the feature vector sequence ofthe new unregistered word are newly stored into the feature vectorbuffer 28, the clustering unit 29 updates the score sheet as representedby broken lines in FIG. 8.

More specifically, the ID, the phoneme sequence, the cluster number, andthe representative member ID of the new unregistered word, and also thescores (s(N+1, 1), s(N+1, 2), . . . , s(N+1, N) shown in FIG. 8) of theunregistered word with respect to the already-stored unregistered wordsare added to the score sheet. Furthermore, the scores (s (1, N+1), s(2,N+1), . . . , s(N, N+1) shown in FIG. 8) of the respectivealready-stored unregistered words with respect to the new unregisteredword are added to the score sheet. Thereafter, as will be describedlater, cluster numbers and representative member IDs described in thescore sheet are updated as required.

In the example shown in FIG. 8, s(i, j) denotes the score of anunregistered word (utterance of an unregistered word) having an ID of iwith respect to an unregistered word (phoneme sequence of anunregistered word) having an ID of j.

In the score sheet (FIG. 8), the score, s(i, i), of an unregistered word(utterance of an unregistered word) having an ID of i with respect tothe unregistered word (phoneme sequence of the unregistered word) havingthe ID of i is also registered. This score s(i, i) is calculated by thematching unit 23 when the matching unit 23 detects the phoneme sequenceof the unregistered word, and thus the clustering unit 29 does not needto calculate this score.

Referring again to FIG. 4, the maintenance unit 31 updates the worddictionary stored in the dictionary memory 25 on the basis of theupdated score sheet stored in the score sheet memory 30.

Herein, the representative member of a cluster is determined as follows.For example, of unregistered words included as members in a cluster, anunregistered word which is highest in the sum of scores with respect tothe other unregistered words (or which is highest in mean score obtainedby dividing the sum of scores by the number of the other unregisteredwords) is selected as the representative member of that cluster. Thatis, if the member ID of a member belonging to a cluster is denoted by k,a member having an ID of K (∈k) given by the following equation isselected as the representative member.K=max_(K) {Σs(k′, k)}  (1)

In equation (1), max_(K){ }denotes k which gives a maximum value to thevalue enclosed in braces { }, k′ denotes, as with k, an ID of a memberof the cluster, and Σ denotes the sum taken by changing k′ over all IDsof the cluster.

In the process of determining the representative member in theabove-described manner, if a cluster includes one or two unregisteredwords as its members, the representative member can be determinedwithout having to calculate the scores. That is, in a case in which acluster includes only one member of an unregistered word, this oneunregistered word is selected as the representative member. On the otherhand, when a cluster includes only two unregistered words as members,either one of the two unregistered words may be selected as therepresentative member.

The method of determining the representative member is not limited tothat described above. For example, of unregistered words included asmembers in a cluster, an unregistered word which is smallest in the sumof distances in the feature vector space from that unregistered word tothe other unregistered words may be selected as the representativemember of that cluster.

The voice recognition unit 50A constructed in the above-described mannerperforms a voice recognition process for recognizing a voice input tothe microphone 15 and performs unregistered word processing onunregistered words.

First, with reference to a flow chart shown in FIG. 9, the voicerecognition process is described.

If a user utters, the uttered voice is input to the microphone 15 andconverted into digital voice data by the AD converter 21. The resultantdigital voice data is supplied to the feature extraction unit 22. Thefeature extraction unit 22 performs, in step S1, acoustic analysis onthe received voice data on a frame-by-frame basis, wherein units offrames are properly determined. The feature extraction unit 22 extractsfeature vector via the above acoustic analysis and supplies the obtainedfeature vector sequence to the matching unit 23 and the unregisteredword period processing unit 27.

In step S2, the matching unit 23 calculates the scores for the featurevector sequence supplied from the feature extraction unit 23, in theabove-described manner. Thereafter, the process proceeds to step S3. Instep S3, on the basis of the scores obtained via the above scorecalculation, the matching unit 23 determines a sequence of entries ofwords indicating the result of voice recognition and outputs theresultant sequence of entries of words.

In the next step S4, the matching unit 23 determines whether the voiceuttered by the user includes an unregistered word.

If it is determined in step S4 that the voice uttered by the userincludes no unregistered word, that is, if the voice recognition resultis obtained without applying the grammatical rule including unregisteredword, “$pat1=$color1 $garbage $color2;”, the process is ended withoutperforming step S5.

On the other hand, if it is determined in step S4 that the voice utteredby the user include an unregistered word, that is, if the voicerecognition result is obtained by applying the grammatical ruleincluding unregistered word, “$pat1=$color1 $garbage $color2;”, theprocess proceeds to step S5. In step S5, the matching unit 23 detects avoice period of the unregistered word by detecting a voice periodcorresponding to variable $garbage in the grammatical rule includingunregistered word. The matching unit 23 also detects a phoneme sequenceof the unregistered word by detecting a phoneme sequence correspondingto the garbage model indicated by the variable $garbage, wherein thegarbage models allows an arbitrary phonemic transition. The resultantvoice period and the phoneme sequence of the unregistered word aresupplied to the unregistered word period processing unit 27, and theprocess is ended.

The feature vector sequence output from the feature extraction unit 22is temporarily stored in the unregistered word period processing unit27. If the unregistered word period processing unit 27 receives thevoice period and the phoneme sequence of the unregistered word from thematching unit 23, the unregistered word period processing unit 27detects a feature vector sequence of the voice in the voice period.Furthermore, the unregistered word period processing unit 27 assigns anID to the unregistered word (phoneme sequence of the unregistered word)supplied from the matching unit 23 and supplies it, together wit thephoneme sequence of the unregistered word and the feature vectorsequence of the voice period, to the feature vector buffer 28.

After the ID, the phoneme sequence, and the feature vector sequence ofthe new unregistered word are stored in the feature vector buffer 28 inthe above-described manner, unregistered word processing is performed.

FIG. 10 is a flow chart showing the unregistered word processing.

In the first step S11 in the unregistered word processing, theclustering unit 29 reads the ID and the phoneme sequence of the newunregistered word from the feature vector buffer 28. Thereafter, theprocess proceeds to step S12.

In step S12, the clustering unit 29 checks the score sheet stored in thescore sheet memory 30 to determine whether there is a cluster which hasalready been obtained (produced).

If it is determined in step S12 that there is no cluster which hasalready been obtained, that is, if the new unregistered word is a firstunregistered word and the score sheet includes no entry of analready-stored unregistered word, then the process proceeds to step S13.In step S13, The clustering unit 29 newly produces a cluster so as toinclude the unregistered word as a representative member thereof, andthe clustering unit 29 registers information associated with the newcluster and the new unregistered word in the score sheet stored in thescore sheet memory 30 thereby updating the score sheet.

That is, the clustering unit 29 registers, into the score sheet (FIG.8), the ID and the phoneme sequence of the new unregistered word readfrom the feature vector buffer 28. Furthermore, clustering unit 29produces a unique cluster number and registers it in the score sheet asthe cluster number of the new unregistered word. The clustering unit 29registers, in the score sheet, the ID of the new unregistered word asthe representative member ID of the new registered word. Thus, in thiscase, the new unregistered word becomes the representative member of thenew cluster.

In this case, because there is no already-stored unregistered word whosescore with respect to the new unregistered word should be calculated,the score calculation is not performed.

After the completion of step S13, the process proceeds to step S22. Instep S22, on the basis of the score sheet updated in step S13, themaintenance unit 31 updates the word dictionary stored in the dictionarymemory 25, and the process is ended.

That is, in this case, because the new cluster has been produced, themaintenance unit 31 recognizes the newly produced cluster by detectingthe cluster number described in the score sheet. The maintenance unit 31adds an entry corresponding to that cluster to the word dictionarystored in the dictionary memory 25 and registers, as a phoneme sequenceof that entry, the phoneme sequence of the representative member of thenew cluster, that is, in this case, the phoneme sequence of the newunregistered word.

On the other hand, if it is determined in step S12 that there is acluster which has already been obtained, that is, if the newunregistered word is not the first unregistered word and thus the scoresheet (FIG. 8) includes an entry (row) of an already-stored unregisteredword, the process proceeds to step S14. In step S14, the clustering unit29 calculates the score of the new unregistered word with respect to thealready-stored unregistered word and also calculates the score of thealready-stored unregistered word with respect to the new unregisteredword.

More specifically, for example, if there are N already-storedunregistered words respectively assigned IDs from 1 to N, and if the IDof the new unregistered word is given as N+1, the clustering unit 29calculates the scores s(N+1, 1), s(N+1, 2), . . . , s(N+1, N) for thenew unregistered word with respect to the N already-stored unregisteredwords, respectively, as shown in a row represented by a broken line inFIG. 8, and also calculates the scores s(1, N+1), s(2, N+1), . . . ,s(N, N+1) for the N already-stored unregistered words, respectively,associated with the new unregistered word. When the clustering unit 29calculates the above-described scores, the feature vector sequences ofthe new unregistered word and the N already-stored unregistered wordsare needed, and they are obtained by referring to the feature vectorbuffer 28.

The clustering unit 29 adds the calculated scores to the score sheet(FIG. 8) together with the ID and the phoneme sequence of the newunregistered word. Thereafter, the process proceeds to step S15.

In step S15, the clustering unit 29 checks the score sheet (FIG. 8) todetect a cluster including a representative member which gives a highestscore of the new unregistered word with respect to the representativemember, s(N+1, i) (i=1, 2, . . . , N). That is, the clustering unit 29detects already-stored unregistered words employed as representativemembers on the basis of the representative member IDs described in thescore sheet and further detects, on the basis of the scores described inthe score sheet, an already-stored unregistered word employed as arepresentative member which gives a highest score to the newunregistered word. The clustering unit 29 then detects a cluster havinga cluster number corresponding to the detected already-storedunregistered word employed as the representative member of that cluster.

Thereafter, the process proceeds to step S16. In step S16, theclustering unit 29 adds the new unregistered word to the clusterdetected in step S15 (hereinafter, referred to simply as the detectedcluster) as a member thereof. That is, the clustering unit 29 writes thecluster number of the representative member of the detected cluster, asthe cluster number of the new unregistered word, into the score sheet.

In the next step S17, the clustering unit 29 divides the detectedcluster into, for example, two clusters. Thereafter, the processproceeds to step 18. In step S18, the clustering unit 29 determineswhether the detected cluster has been divided, in the cluster divisionprocess performed in step S17, into two clusters. If it is determinedthat the detected cluster has been divided into two clusters, then theprocess proceeds to step S19. In step S19, the clustering unit 29determines the cluster-to-cluster distance between the two clustersobtained by dividing the detected cluster (hereinafter, such twoclusters will be referred to as a first sub-cluster and a secondsub-cluster).

The cluster-to-cluster distance between the first and secondsub-clusters is defined, for example, as follows.

Herein, let k denote the ID of an arbitrary member (unregistered word)of the first and second sub-clusters, and let k1 and k2 denote the IDsof representative members (unregistered words) of the first and secondsub-clusters, respectively. The cluster-to-cluster distance between thefirst and second sub-clusters is given by D(k1, k2) obtained by thefollowing equation:D(k1,k2)=maxal_(k){abs(log(s(k,k1)) −log(s(k,k2})  (2)

In equation (2), abs( ) represent the absolute value of a value enclosedin parentheses ( ), maxval_(K){ } represents the maximum value of valuesobtained by changing k of a variable enclosed in braces { }, and logrepresents a natural or common logarithm.

Herein, if a member whose ID is i is denoted as a member #i, thereciprocal of the score, 1/s(k, k1), corresponds to the distance betweena member #k and a representative member k1, represented by equation (2),and the reciprocal of the score, 1/s(k, k2), corresponds to the distancebetween the member #k and a representative member k2. Thus, according toequation (2) the cluster-to-cluster distance between the first andsecond sub-clusters is given by the maximum difference between thedistance of any member of the first and second sub-clusters relative tothe representative member #k1 of the first sub-cluster and the distanceof any member of the first and second sub-clusters relative to therepresentative member #k2 of the second sub-cluster.

Note that the definition of the cluster-to-cluster distance is notlimited to that described above. For example, the sum of distances inthe feature vector space is determined by means of the DP matchingbetween the representative member of the first sub-cluster and therepresentative member of the second sub-cluster, and the sum ofdistances may be employed as the cluster-to-cluster distance.

After completion of step S19, the process proceeds to step 20. In stepS20, the clustering unit 29 determines whether the cluster-to-clusterdistance between the first and second sub-clusters is greater than (orequal to or greater than) a predetermined threshold ε.

If it is determined in step S20 that the cluster-to-cluster distance isgreater than the predetermined threshold ε, that is, if it is determinedthat a plurality of unregistered words included as members in thedetected cluster should be separated into two clusters from the point ofview of acoustic features thereof, then the process proceeds to stepS21. Instep S21, the clustering unit 29 registers the first and secondsub-clusters into the score sheet stored in the score sheet memory 30.

That is, the clustering unit 29 assigns unique cluster numbers to thefirst and second sub-clusters, respectively, and updates the score sheetsuch that, of the cluster numbers of the respective members of thedetected cluster, the cluster numbers of those members which have beenclustered into the first sub-cluster are replaced with the clusternumbers of the first sub-cluster, and the cluster numbers of thosemembers which have been clustered into the second sub-cluster arereplaced with the cluster numbers of the second sub-cluster.

The clustering unit 29 updates the score sheet such that the ID of therepresentative member of the first sub-cluster is employed as therepresentative member ID of members clustered into the first sub-clusterand the ID of the representative member of the second sub-cluster isemployed as the representative member ID of members clustered into thesecond sub-cluster.

The cluster number of the detected cluster may be employed as that ofeither one of the first or second sub-cluster.

After the clustering unit 29 has registered the first and secondsub-clusters into the score sheet in the above-described manner, theprocess proceeds from step S21 to S22. In step S22, the maintenance unit31 updates the word dictionary stored in the dictionary memory 25 on thebasis of the score sheet. Thereafter, the process is ended.

That is, in this case, because the detected cluster has been dividedinto the first and second sub-clusters, the maintenance unit 31 firstdeletes the entry corresponding to the detected cluster from the worddictionary. The maintenance unit 31 then adds two entries correspondingto the first and second sub-clusters to the word dictionary andregisters the phoneme sequence of the representative member of the firstsub-cluster as the phoneme sequence of the entry corresponding to thefirst sub-cluster and the phoneme sequence of the representative memberof the second sub-cluster as the phoneme sequence of the entrycorresponding to the second sub-cluster.

On the other hand, in a case in which it is determined in step S18 thatthe detected cluster was not divided into two sub-clusters in thecluster division process in step S17, or in a case in which it isdetermined in step S20 that the cluster-to-cluster distance between thefirst and second sub-clusters is not greater than the predeterminedthreshold E (that is, in a case in which the acoustic features of theunregistered words included as members in the detected cluster aredetermined to be not so different that the detected cluster should bedivided into the first and second sub-clusters), the process proceeds tostep S23. In step S23, the clustering unit 29 determines a newrepresentative member of the detected cluster and updates the scoresheet.

That is, the clustering unit 29 refers to the score sheet stored in thescore sheet memory 30 to detect scores s(k′, k) needed in calculation ofequation (1) for each member of the detected cluster to which the newunregistered word has been added as a member thereof. The clusteringunit 29 determines the ID of a member to be employed as the newrepresentative member of the detected cluster in accordance withequation (1) using the detected scores s(k′, k) The clustering unit 29then rewrites the score sheet (FIG. 8) such that the representativemember ID of each member of the detected cluster is replaced with the IDof the new representative member of the detected cluster.

Thereafter, the process proceeds to step S22. In step S22, themaintenance unit 31 updates the word dictionary stored in the dictionarymemory 25 on the basis of the score sheet, and the process is ended.

That is, in this case, the maintenance unit 31 refers to the score sheetto detect the new representative member of the detected cluster andfurther detect the phoneme sequence of that representative member. Themaintenance unit 31 then updates the word dictionary by replacing thephoneme sequence of the entry corresponding to the detected cluster withthe phoneme sequence of the new representative member of the detectedcluster.

Referring now to a flow chart shown in FIG. 11, the cluster divisionprocess in step S17 shown in FIG. 10 is described in further detailbelow.

In the cluster division process, first in step S31, the clustering unit29 selects a combination of arbitrary two members, which have not beenselected yet, from the detected cluster to which the new unregisteredword has been added as a member, and the clustering unit 29 employsthose two members as provisional representative members. Hereinafter,those two provisional representative members will be referred to as afirst provisional representative member and a second provisionalrepresentative member, respectively.

In the next step S32, the clustering unit 29 determines whether themembers of the detected cluster can be divided into two sub-clusterssuch that the first and second provisional representative members becomethe representative members of the respective two sub-clusters.

In order to determine whether the first and second provisionalrepresentative members can be the representative members, it isnecessary to calculate equation (1), wherein the scores s(k′, k) used inthe calculation can be obtained by referring to the score sheet.

In a case in which it is determined in step S32 that the members of thedetected cluster cannot be divided into two sub-clusters such that thefirst and second provisional representative members become therepresentative members of the respective two sub-clusters, the processjumps to step S34 without performing step S33.

However, if it is determined in step S32 that the members of thedetected cluster can be divided into two sub-clusters such that thefirst and second provisional representative members become therepresentative members of the respective two sub-clusters, the processproceeds to step S33. In step S33, the clustering unit 29 divides themembers of the detected cluster into two sub-clusters such that thefirst and second provisional representative members become therepresentative members of the respective two sub-clusters, and theclustering unit 29 employs the combination of the two sub-clusters ascandidates for the final first and second sub-clusters into which thedetected cluster should be divided (hereinafter, such candidates will bereferred to as a combination of candidate sub-clusters). Thereafter, theprocess proceeds to step S34.

In step S34, the clustering unit 29 determines whether the detectedcluster includes a combination of two members which has not yet beenselected as a combination of first and second provisional representativemembers. If there is such a combination of two members, the processreturns to step S31 to select a combination of two members, which hasnot yet been selected as a combination of first and second provisionalrepresentative members, from the members of the detected cluster.Thereafter, the process is performed in a similar manner as describedabove.

On the other hand, if it is determined in step S34 that there is no sucha combination of two members of the detected cluster which has not yetbeen selected as a combination of first and second provisionalrepresentative members, the process proceeds to step S35. In step S35,the clustering unit 29 determines whether there is a combination ofcandidate sub-clusters.

If it is determined in step S35 that there is no combination ofcandidate sub-clusters, the flow exits from the process withoutperforming step S36. In this case, it is determined in step S18 in FIG.10 that the detected cluster cannot be divided.

On the other hand, if it is determined in step S35 that there is acombination of candidate sub-clusters, the process proceeds to step S36.In step S36, if there is a plurality of combinations of candidatesub-clusters, the clustering unit 29 determines the cluster-to-clusterdistance between the two candidate sub-clusters for each combination ofcandidate sub-clusters. The clustering unit 29 then determines acombination of candidate sub-clusters having a minimumcluster-to-cluster distance and clustering unit 29 returns thatcombination of candidate sub-clusters as a result of division of thedetected cluster, that is, as the first and second sub-clusters. In acase in which there is only one combination of candidate sub-clusters,those candidate sub-clusters are employed as the first and secondsub-clusters.

In this case, it is determined in step S18 in FIG. 10 that the detectedcluster has been divided.

As described above, the clustering unit 29 detects, from the existingclusters each including an unregistered word, a cluster (detectedcluster) to which the new unregistered word is to be added as a memberthereof, and the clustering unit 29 divides the detected cluster on thebasis of the members of the detected cluster so that the newunregistered word becomes a new member of the detected cluster, therebyeasily clustering the unregistered words such that unregistered wordshaving similar acoustic features are clustered into the same cluster.

Furthermore, the maintenance unit 31 updates the word dictionary on thebasis of the result of the clustering, and thus it becomes possible toeasily register an unregistered word into the word dictionary withoutcausing a great increase in the size of the word dictionary.

Furthermore, even if the matching unit 23 detects a wrong voice periodfor an unregistered word, such an unregistered word is clustered, in theprocess of dividing the detected cluster, into a cluster different froma cluster into which an unregistered word whose voice period has beencorrectly detected. Although an entry corresponding to the cluster inwhich the unregistered word has been improperly clustered is registeredin the dictionary, the phoneme sequence never have a large contributionto the scores in voice recognition performed after that, because thephoneme sequence of that entry corresponds to the voice period which wasnot correctly detected. Therefore, even if a wrong detection occurs fora voice period of an unregistered word, the wrong detection does nothave a significant influence on the following voice recognition.

FIG. 12 shows a result of a simulation of clustering for an utteranceincluding an unregistered word. In FIG. 12, each entry (row) representsone cluster. In FIG. 12, a phoneme sequence of a representative member(unregistered word) of each cluster is described on the left-hand side,and an utterance of the unregistered word included as a member in thecluster and the number of utterances are described on the right-handside.

More specifically, in FIG. 12, for example, in the entry in the firstrow, described is a cluster including, as only one member, an utteranceof an unregistered word “

” (a Japanese word pronounced as “furo” and corresponding to an Englishword “bath”), and “doroa:” is obtained as the phoneme sequence of therepresentative member. As another example, in the entry in the secondrow, there is described a cluster including, as members, threeutterances of the unregistered word “

” (bath), and “kuro” is obtained as the phoneme sequence of therepresentative member.

As still another example, in the entry in the seventh row, there isdescribed a cluster including, as members, four utterances of anunregistered word “

” (a Japanese word pronounced as “hon” and corresponding to an Englishword “book”), and “NhoNde:su” is obtained as the phoneme sequence of therepresentative member. In the entry in the eighth row, there isdescribed a cluster including, as members, one utterance of anunregistered word “

” a (Japanese word pronounced as “orenji” and corresponding to anEnglish word “orange”) and nineteen utterances of the unregistered word“

” (book), and “ohoN” is obtained as the phoneme sequence of therepresentative member. The other entries are also described in a similarmanner.

As can be seen from FIG. 12, good results are obtained in clustering forutterances of the same unregistered word.

In the entry in the eighth row in FIG. 12, one utterance of theunregistered word “

” (orange) and nineteen utterances of the unregistered word “

” (book) are clustered into the same cluster. Judging from theutterances included as members in this cluster, this cluster should beof the unregistered word “

” (book). However, the cluster also includes as a member an utterance ofthe unregistered word “

” (orange). If utterances of the unregistered word “

” (book) are further input, then this cluster will be divided into acluster including, as members, only utterances of the unregistered word“

” (book) and a cluster including, as a member, only the utterance of theunregistered word “

” (orange).

Although the present invention has been described above with referenceto embodiments in which the invention is applied to the entertainmentrobot (pet robot), the invention can also be applied to a wide varietyof apparatuses or systems in which a voice recognition apparatus isused, such as a voice interactive system. Furthermore, the presentinvention can be applied not only to actual robots that act in the realworld but also to virtual robots such as that displayed on a displaysuch as a liquid crystal display.

In the first embodiment described above, a sequence of processing isperformed by executing the program using the CPU 10A. Alternatively, thesequence of processing may also be performed by dedicated hardware.

The program may be stored, in advance, in the, memory 10B (FIG. 2).Alternatively, the program may be stored (recorded) temporarily orpermanently on a removable storage medium such as a flexible, a CD-ROM(Compact Disc Read Only Memory), an MO (Magnetooptical) disk, a DVD(Digital Versatile Disc), a magnetic disk, or a semiconductor memory. Aremovable storage medium on which the program is stored may be providedas so-called packaged software thereby allowing the program to beinstalled on the robot (memory 10B).

The program may also be installed into the memory 10B by downloading theprogram from a site via a digital broadcasting satellite and via awireless or cable network such as a LAN (Local Area Network) or theInternet.

In this case, when the program is upgraded, the upgraded program may beeasily installed in the memory 10B.

In the examples described above, the processing steps described in theprogram to be executed by the CPU 10A for performing various kinds ofprocessing are not necessarily required to be executed in time sequenceaccording to the order described in the flow chart. Instead, theprocessing steps may be performed in parallel or separately (by means ofparallel processing or object processing).

The program may be executed either by a single CPU or by a plurality ofCPUs in a distributed fashion.

The voice recognition unit 50A shown in FIG. 4 maybe realized by meansof dedicated hardware or by means of software. When the voicerecognition unit 50A is realized by software, a software program isinstalled on a general-purpose computer or the like.

FIG. 13 illustrates an example of the configuration of a computer onwhich the program used to realize the voice recognition unit 50A isinstalled.

That is, another embodiment of a voice recognition apparatus 91according to the present invention is shown in FIG. 13.

As shown in FIG. 13, the program may be stored, in advance, on a harddisk 105 serving as a storage medium or in a ROM 103 which are disposedinside the computer.

Alternatively, the program may be stored (recorded) temporarily orpermanently on a removable storage medium 111 such as a flexible disk, aCD-ROM, an MO disk, a DVD, a magnetic disk, or a semiconductor memory.Such a removable storage medium 111 may be provided in the form ofso-called package software.

Instead of installing the program from the removable storage medium 111onto the computer, the program may also be transferred to the computerfrom a download site via a digital broadcasting satellite by means ofwireless transmission or via a network such as an LAN (Local AreaNetwork) or the Internet by means of cable communication. In this case,the computer receives, using a communication unit 108, the programtransmitted in the above-described manner and installs the receivedprogram on the hard disk 105 disposed in the computer.

The voice recognition apparatus 91 includes a CPU (Central ProcessingUnit) 102. The CPU 102 is connected to an input/output interface 110 viaa bus 101 so that when a command issued by operating an input unit 107including a keyboard, a mouse, a microphone, and an AD converter isinput via the input/output interface 110, the CPU 102 executes theprogram stored in a ROM (Read Only Memory) 103 in response to thecommand. Alternatively, the CPU 102 may execute a program loaded in aRAM (Random Access Memory) 104 wherein the program may be loaded intothe RAM 104 by transferring a program stored on the hard disk 105 intothe RAM 104, or transferring a program which has been installed on thehard disk 105 after being received from a satellite or a network via thecommunication unit 108, or transferring a program which has beeninstalled on the hard disk 105 after being read from a removablerecording medium 111 loaded on a drive 109. By executing the program,the CPU 102 performs the process described above with reference to theflow chart or the process described above with reference to the blockdiagrams. The CPU 102 outputs the result of the process, as required, toan output unit 106 including an LCD (Liquid Crystal Display), a speaker,and a DA (Digital Analog) converter via the input/output interface 110.The result of the process may also be transmitted via the communicationunit 108 or may be stored on the hard disk 105.

FIG. 14 shows an example of the configuration of the software program ofthe voice recognition apparatus 91. This software program includes aplurality of modules. Each module has its own independent algorithm, andeach module executes a particular operation in accordance its algorithm.Each module is stored in the RAM 13 and read and executed by the CPU 11.

The respective modules shown in FIG. 14 correspond to the blocks shownin FIG. 4. More specifically, an acoustic model buffer 133 correspondsto the acoustic model memory 24, a dictionary buffer 134 to thedictionary memory 25, a grammar buffer 135 to the grammar memory 26, afeature extraction module 131 to the feature extraction unit 22, amatching module 132 to the matching unit 23, an unregistered word periodprocessing module 136 to the unregistered word period processing unit27, a feature vector buffer 137 to the feature vector buffer 28, aclustering module 138 to the clustering unit 29, a score sheet buffer139 to the score sheet memory 30, and a maintenance module 140 to themaintenance unit 31.

Note that in the present example, in the input unit 107 shown in FIG.13, an input analog voice signal obtained via the microphone is suppliedto the AD converter, which converts the input analog voice signal todigital voice data by means of an A/D (Analog/Digital) conversionincluding sampling and quantizing, and the resultant digital voice datais supplied to the feature extraction module 131.

In the present example, the feature vector buffer 137 stores, forexample, as shown in FIG. 15, an ID, a phoneme sequence, a featurevector sequence, and a storage time of an unregistered word suppliedfrom the unregistered word period processing module 136 such that theyare related to each other. In other words, the feature vector buffer 137stores a set of data indicating entries (in rows) corresponding torespective unregistered words.

In the example shown in FIG. 15, sequential numerals starting from 1 areassigned to respective unregistered words. Therefore, for example, whenIDs, phoneme sequences, feature vector sequences, and storage times forN unregistered words have been stored in the feature vector buffer 137,if the matching module 132 detects a voice period and a phoneme sequenceof another unregistered word, the unregistered word period processingmodule 136 assigns N+1 as an ID to the detected unregistered word, andthe feature vector buffer 137 stores the ID (N+1), the phoneme sequence,the feature vector sequence, and the storage time of that unregisteredword as represented by broken lines in FIG. 15.

The entries shown in FIG. 15 are similar to the respective entries shownin FIG. 7 except that each entry shown in FIG. 15 includes additionaldata indicating a storage time. Each storage time indicates a time atwhich an entry is stored (recorded) into the feature vector buffer 137.The method of using the storage time will be described later.

As will be described later, when the clustering module 138 performsclustering on a new unregistered word, the clustering module 138 refersto the “feature vectors” stored in the feature vector buffer 137.Hereinafter, such “voice information”, which is referred to whenclustering is performed on an unregistered word, will be called“utterance information”.

That is, the “utterance information” is not limited to “feature vectors”but a “PCM (Pulse Code Modulation) signal” such as voice data suppliedto the feature extraction module 131 may also be employed as utteranceinformation. In this case, the feature vector buffer 137 stores the “PCMsignal” instead of the “feature vector sequence”.

Thus, the voice recognition apparatus 91 formed of the modules describedabove can operate in a similar manner as the voice recognition unit 50Ashown in FIG. 4, although the detailed structure and operation of themodules corresponding to the voice recognition unit 50A are notdescribed herein.

The voice recognition unit 50A needs to store voice waveforms (forexample, digital voice data) of clusters of unregistered words orfeature vectors (for example, MFCCs (Mel Frequency CepstrumCoefficients) obtained by performing MFCC analysis on digital voicedata) as utterance information in a particular memory area or thefeature vector buffer 28 serving as a memory so that the utteranceinformation can be used to perform clustering on a newly inputunregistered word.

That is, in the above-described process, when the voice recognition unit50A detects a cluster to which an unregistered word is to be added as anew member, from existing clusters obtained by performing clustering onvoices, the voice recognition unit 50A refers to the past utteranceinformation stored in the particular storage area or the feature vectorbuffer 28 serving as the memory.

If the voice recognition unit 50A stores utterance informationcorresponding to all unregistered words one after another, then a largestorage area or memory area is consumed with increasing number of inputunregistered words (with increasing number of acquired unregisteredwords).

In the embodiment shown in FIG. 14, to avoid the above problem, there isadditionally provided a feature vector deletion module 141 for, when apredetermined condition is satisfied, deleting particular utteranceinformation and associated data from the feature vector buffer 137.

More specifically, for example, the feature vector deletion module 141checks a score sheet, similar to that shown in FIG. 8, stored in thescore sheet buffer 139 to determine whether the number of membersbelonging to a particular cluster has exceeded a predetermined firstnumber. If it is determined that the number of such members is greaterthan the predetermined first number, the feature vector deletion module141 deletes utterance information and associated data of a second numberof members of the members belonging to the particular cluster from thefeature vector buffer 137. Herein, data associated with a memberincludes an ID and a phoneme sequence of that member and also data ofthe member described on the score sheet.

Thus, the feature vector deletion module 141 prevents clusters frombecoming greater than a predetermined size, thereby not only suppressingconsumption of memory (such as the RAM 103) but also preventing areduction in operation speed of the voice recognition apparatus 91, thatis, preventing degradation in performance of the voice recognitionapparatus 91.

Note that the first and second numbers described above are selected suchthat the first number is equal to or greater than the second number. Thesecond number of members to be deleted may be selected, for example, inthe order of storage time from oldest to newest according to the storagetime data shown in FIG. 15.

Furthermore, if the feature vector deletion module 141 determines thatdata supplied from the not-referred-to time calculation module 142indicates that a particular cluster has not been referred to at all overa period of time equal to or longer than a predetermined length, thefeature vector deletion module 141 deletes utterance information andassociated data of members of that particular cluster from the featurevector buffer 137.

More specifically, for example, the not-referred-to time calculationmodule 142 checks the feature vector buffer 137 to detect the latest oneof times (storage times shown in FIG. 15) at which utterance informationof respective members belonging to a particular cluster were stored intothe feature vector buffer 137 (that is, the not-referred-to timecalculation module 142 detects the time at which an entry of anunregistered word clustered last into the particular cluster was storedinto the feature vector buffer 137), and the not-referred-to timecalculation module 142 employs the detected latest time as the lastreference time of that particular cluster.

The not-referred-to time calculation module 142 then subtracts thedetected last reference time from the current time to determine thenot-referred-to time during which the particular cluster has not beenreferred to and supplies the not-referred-to time to the feature vectordeletion module 141.

Although in the present embodiment, the not-referred-to time calculationmodule 142 calculates the not-referred-to time for all clusters atpredetermined time intervals, there is no limitation on the number ofclusters the not-referred-to time of which is calculated. For example,the not-referred-to time calculation module 142 may calculate thenot-referred-to time only for clusters specified by a user.

Furthermore, the method of the calculation performed by thenot-referred-to time calculation module 142 is not limited to thatdescribed above. For example, although in the above example thenot-referred-to time is calculated on the basis of the storage timesstored in the feature vector buffer 137, the storage times are notnecessarily needed to be stored in the feature vector buffer 137, butthe not-referred-to time calculation module 142 may directly detect andstore the last reference time of a particular cluster and may calculatethe not-referred-to time on the basis of the stored last reference time.

In the above example, on the basis of the not-referred-to time suppliedfrom the not-referred-to time calculation module 142, the feature vectordeletion module 141 deletes utterance information and associated data ofall members belonging to a cluster to which no new member has not beenregistered for a long time, from the feature vector buffer 137.Alternatively, instead of deleting all member of such a cluster,utterance information and associated data of only some members of thecluster may be deleted.

Furthermore, although in the example described above, the storage timeof a member (unregistered word) registered last in a cluster is employedas the last reference time of that cluster, the last reference time of acluster may be determined in a different manner. For example, a time atwhich a cluster is detected in step S15 in FIG. 10, or a time at whichsub-clusters are registered in step S21, or a time at which a cluster isreferred to in some process may be employed as the last reference time.

When the feature vector deletion module 141 receives a delete command(trigger signal) indicating that a particular cluster should be deletedvia the input unit 107 (for example, a keyboard) the feature vectordeletion module 141 may delete utterance information and associated dataof part or all of members belonging to that particular cluster from thefeature vector buffer 137.

When the voice recognition apparatus 91 is disposed on the pet robotshown in FIG. 1, if the feature vector deletion module 141 deletes aparticular feature vector sequence in response to not an internal stateof the voice recognition apparatus 91 but an external stimulus, amnesiadue to a strong stimulus can be realized in the robot.

Furthermore, for example, if the level of emotion (emotion level)supplied from the emotion control module 143 is greater than apredetermined value (level), the feature vector deletion module 141 maydelete utterance information and associated data of part or all ofmembers belonging to a particular cluster from the feature vector buffer137.

In a case in which the voice recognition apparatus 91 is provided in therobot shown in FIG. 1, the information control module 143 can berealized by the model memory 51 shown in FIG. 3. That is, in this case,the model memory 51 supplies information indicating an emotion level asstate information to the feature vector deletion module 141, wherein theemotion level indicates the state of emotion, instinct, or growthindicated by a value of the emotion model, the instinct model, and thegrowth model.

If the feature vector deletion module 141 deletes particular utteranceinformation stored in the feature vector buffer 137 on the basis of theemotion level (parameter value of emotion (value of a model)) suppliedfrom the emotion control module 143, it is possible to realize a loss inmemory in a robot in response to an occurrence of strong angry or thelike in the robot shown in FIG. 1 (in response to an increase in theparameter associated with the “angry” beyond a predetermined value).

When the data supplied from the used-memory-area calculation module 144indicates that the total amount of used memory space (for example, thememory space of the RAM 103 or the like, shown in FIG. 1, including thefeature vector buffer 137 and the score sheet buffer 139) has exceeded apredetermined value, the feature vector deletion module 141 may deleteutterance information and associated data of part or all of membersbelonging to a particular cluster from the feature vector buffer 137.

More specifically, the used-memory-area calculation module 144 alwayscalculates the total amount of used memory space (total memoryconsumption amount) and supplies data indicating the total amount ofused memory space to the feature vector deletion module 141 atpredetermined time intervals.

As described above, the feature vector deletion module 141 alwaysmonitors the amount of memory consumption (of the RAM 103 or the like),and if the amount of memory consumption has exceeded the predeterminedvalue, the feature vector deletion module 141 reduce the amount ofmemory consumption by deleting utterance information and associated dataof members belonging to a cluster stored in the feature vector buffer137, thereby not only suppressing consumption of memory (such as the RAM103) but also preventing a reduction in operation speed of the voicerecognition apparatus 91, that is, preventing degradation in performanceof the voice recognition apparatus 91.

Although in the example described above, the feature vector deletionmodule 141 determines whether the value of a parameter has exceeded apredetermined value, wherein the parameter may be the number of membersof a cluster (the number of entries associated with the respectivemembers of the same cluster stored in the feature vector buffer 137),the not-referred-to time indicated by the not-referred-to timecalculation module 142, the emotion level indicated by the emotioncontrol module 143, or the memory consumption amount indicated by theused-memory-area calculation module 144, and if the feature vectordeletion module 141 determines that the value of the parameter hasexceeded the predetermined value, then the feature vector deletionmodule 141 determines that the predetermined condition is satisfied andthe feature vector deletion module 141 deletes part or all of themembers of the cluster, the method of deleting members (utteranceinformation and associated data of members) is not limited to such amethod.

For example, instead of making the judgment on the parameter value, thefeature vector deletion module 141 may determine that the particularcondition is satisfied when a trigger signal (such as a delete commandsupplied via the input unit 107) is input, and may delete particularutterance information.

In this case, the emotion control module 143, the not-referred-to timecalculation module 142, and the used-memory-area calculation module 144may make the above-described judgment on the parameter value, and if itis determined in the judgment process that the value of the parameter(the emotion level, the not-referred-to time, or the total amount ofused memory space) associated with one of these modules is greater thanthe predetermined threshold, a trigger signal may be supplied to thefeature vector deletion module 141.

The trigger signal supplied to the feature vector deletion module 141 isnot limited to that described above, but another trigger signal such asthat generated by a user when an arbitrary condition is satisfied mayalso be employed.

As described above, when the feature vector deletion module 141determines that the particular condition is satisfied, the featurevector deletion module 141 deletes particular utterance information ofmembers stored in the feature vector buffer 137. The utteranceinformation deleted in the above process may be arbitrarily selected(set), and the number of pieces of utterance information deleted mayalso be arbitrarily selected (set). For example, a user or amanufacturer may set the conditions associated with deletion ofutterance information.

In order to maintain the voice recognition accuracy of the voicerecognition apparatus 91 at a high level without encountering areduction in performance thereof, it is desirable to preferentiallydelete such members described below.

That is, in a case in which part of members of a cluster is deleted, ifa representative member of that cluster or a member having a rathersmall distance relative to the representative member (that is, a memberhaving a large score associated with the representative member) isdeleted, a great change will occur in the structure itself of thatcluster. Therefore, it is desirable to preferentially delete a memberother than such the members described above.

On the other hand, members of a cluster including a small number ofcluster, members having a large distance relative to a representativemember, and members of a cluster to which a new member has not beenadded for a long time can be regarded as not having a large influence onthe voice recognition accuracy, and thus it is desirable topreferentially delete such members.

When the feature vector deletion module 141 deletes utteranceinformation and associated data of members stored in the feature vectorbuffer 137, the score sheet stored in the score sheet buffer 139includes data associated with the deleted members.

Therefore, if the feature vector deletion module 141 deletes utteranceinformation and associated data stored in the feature vector buffer 137,the feature vector deletion module 141 also deletes, from the scoresheet, data associated with the members deleted from the feature vectorbuffer 137.

For example, if the feature vector deletion module 141 deletes data (ID,phoneme sequence, feature vector sequence (utterance information), andstorage time) of an entry (in a row) of an ID of 3 shown in FIG. 15,then the feature vector deletion module 141 further deletes, from thescore sheet shown in FIG. 8, data (ID, phoneme sequence, cluster number,representative member ID, and scores s(3, i) (i=1, . . . , N+1)) of anentry (in a row) of an ID of 3 and also deletes scores s(j, 3) (j=1, . .. , N+1) of member having the other IDs associated with the member ofthe ID of 3.

In this case, the clustering module 138 reselects (redetermines) arepresentative member of a cluster to which the deleted member belonged.In the specific example described above, the representative member ofthe cluster (with a cluster number of 1) to which the member having theID of 3 belonged as shown in FIG. 8 is reselected (redetermined). If thereselected representative member is different from the previous one(that is, if a member having an ID other than 1 is reselected as arepresentative member), a possibility occurs that the structures of allclusters are changed. Thus, reclustering is performed on unregisteredwords having any ID.

The method of reclustering is not limited to a specific one, and, forexample, a k-means method may be employed.

In this case, the clustering module 138 performs processes (1) to (3)described below, wherein it is assumed that N unregistered words areregistered in the score sheet stored in the score sheet buffer 139 andthese unregistered words are clustered into k clusters.

(1) Of the N unregistered words, arbitrary K unregistered wordsclustered into k clusters such that each of K unregistered words becomesan initial cluster center and the initial cluster center is employed asthe provisional representative member.

(2) The scores of all data (N unregistered words) associated with the krepresentative members are recalculated, and each of the N unregisteredwords is registered as a member of a cluster including a representativemember giving a highest recalculated score to the unregistered word.

(3) A representative member is selected for each of the k clusters intowhich members have been newly registered.

In the process (2) described above, the scores can be determined byreferring to the score sheet without actually calculating the score.However, the clustering module 138 may actually calculate the scores inthe process (2). In this case, utterance information of the Nunregistered words is needed, wherein the unregistered word can beobtained by referring to the feature vector buffer 137.

When the clustering module 138 actually calculates the scores, if notfeature vector sequences but PCM signals (voice data) are stored asutterance information in the feature vector buffer 137, the clusteringmodule 138 calculates the scores on the basis of the PCM signals.

If, as a result of the reclustering using the k-means method, thestructure of a cluster other than a cluster to which a deletedunregistered word previously belonged has been changed, the deletion ofthat unregistered word is regarded as having a large influence, and theclustering module 138 and the feature vector deletion module 141 cancelthe deletion of that unregistered word and also cancel all processes(updating of the score sheet and the reclustering) a rising from thedeletion so as to make associated data return back into the state inwhich the associated data was before the deletion was performed (byperforming the undo process)

Referring now to a flow chart shown in FIG. 16, the voice recognitionprocess performed by the voice recognition apparatus 91 shown in FIG. 14is described.

In this example described below, it is assumed that the data shown inFIG. 15 is stored in the feature vector buffer 137, and the score sheetshown in FIG. 8 is stored in the score sheet buffer 139. Furthermore, itis assumed that utterance information is represented in the form of afeature vector sequence.

In step S101, the feature vector deletion module 141 determines whethera command for deleting an unregistered word has been issued.

In the present example, the feature vector deletion module 141determines that the command for deleting an unregistered word has beenissued, when one of conditions (1) to (5) described below is satisfied:

(1) if the number of members belonging to particular clusters of theclusters registered in the score sheet stored in the score sheet buffer139 becomes greater than a predetermined number;

(2) if the data supplied from the not-referred-to time calculationmodule 142 indicates that the not-referred-to time of a particularcluster becomes longer than a predetermined time;

(3) if a delete command (trigger signal) is supplied via the input unit107;

(4) if the value of a parameter of emotion (emotion level) supplied fromthe emotion control module 143 is greater than a predetermined value(level); or

(5) if the data supplied from the used-memory-area calculation module144 indicates that the total amount of used memory space (of the RAM 103or the like) is greater than a predetermined value.

If the feature vector deletion module 141 determines in step S101 thatthe command for deleting an unregistered word has been issued, then thefeature vector deletion module 141 executes, in the next step S102, an“unregistered word deletion routine” on the unregistered word commandedto be deleted (hereinafter, referred to as the unregistered word to bedeleted). Thereafter, the process returns to step S101 to againdetermine whether an unregistered word is commanded to be deleted.

The details of the “unregistered word deletion process” are described inFIG. 17. Referring to FIG. 17, the “unregistered word deletion process”is described below.

First, in step S121, the feature vector deletion module 141 deletes, ofthe data stored in the feature vector buffer 137, such datacorresponding to an unregistered word to be deleted.

For example, in a case in which an unregistered word of an ID of 3 shownin FIG. 15 is to be deleted, of the data shown in FIG. 15, data (ID,phoneme sequence, feature vector (utterance information), and storagetime) of an entry (in a row) of the ID of 3 is deleted.

Instep S122, the feature vector deletion module 141 corrects the scoresheet stored in the score sheet buffer 139.

For example, if the data of the entry of the ID of 3 has been deleted instep S121 in the above-described manner, then, in step S122, of the datastored in the score sheet shown in FIG. 8, data (ID, phoneme sequence,cluster number, representative member ID, and scores s(3, i) (i=1, . . ., N+1)) of an entry (in a row) of the ID of 3 is deleted, and scoress(j, 3) (j=1, . . . , N+1) of unregistered words having IDs other than 3associated with the deleted unregistered word having the ID of 3 aredeleted.

In step S123, the clustering module 138 reselects (redetermines) arepresentative member for the cluster to which the deleted unregisteredword previously belonged.

In this specific example, the unregistered word having the ID of 3 wasdeleted, and thus a representative member for a cluster having a clusternumber of 1 (to which the unregistered word having the ID of 3previously belonged) described in the score sheet shown in FIG. 8 isreselected in the above-described manner.

In step S124, the clustering module 138 determines whether thereselected representative member is different from the previousrepresentative member (that is, whether the representative memberreselected in step S123 is different from the immediately previousrepresentative member). If no change is detected in the representativemember, the flow returns. That is, step S102 in FIG. 16 is ended and theprocess returns to step S101 to repeat the process from step S101.

For example, in a case in which a member having an ID of 1 wasreselected in step S123 as the representative member, it is determinedthat no change has occurred in the representative member. On the otherhand, in a case in which a member having an ID other than 1 wasreselected as the representative member, it is determined that therepresentative member has been changed.

In a case in which the clustering module 138 has determined in step S124that the representative member has been changed, the process proceeds tostep S125 in which the clustering module 138 performs reclustering onall unregistered words (in the present example, all unregistered wordsregistered in the score sheet shown in FIG. 8 except for theunregistered word having the ID of 3). That is, the clustering module138 performs reclustering on all unregistered words by means of, forexample, the k-means method described earlier.

In step S126, the clustering module 138 determines whether a change hasoccurred in the structure of a cluster other than the cluster to whichthe deleted unregistered word previously belonged (more specifically,for example, clustering module 138 determines whether a change in termsof members belonging to a cluster has occurred or whether arepresentative member of a cluster has been changed to another member).If it is determined that no change has occurred in cluster structure,the process proceeds to step S128. In step S128, the maintenance module140 updates the word dictionary stored in the dictionary buffer 134 onthe basis of the score sheet updated (corrected) in step S122.Thereafter, the flow returns.

That is, in the present case, when a new representative member for thecluster to which the deleted unregistered word previously belonged isreselected (in step S123), the new representative member is differentfrom the previous representative member (step S124), and thusmaintenance module 140 refers to the score sheet to detect clusterswhose representative members have been redetermined. The maintenancemodule 140 registers, in the word dictionary stored in the dictionarybuffer 134, the phoneme sequences of the new representative members asthe phoneme sequences of entries corresponding to the clusters whoserepresentative member has been redetermined.

On the other hand, if the clustering module 138 determines in step S126that a change has occurred in cluster structure, then, in step S127, theclustering module 138 and the feature vector deletion module 141 returnthe contents of the feature vector buffer 137 and the score sheet buffer139 into the states in which they were before deletion was performed (instep S121). That is, the clustering module 138 and the feature vectordeletion module 141 perform the undo process until the state, in whichassociated data was before the unregistered words were deleted, isreached. Thereafter, the flow returns.

The process (undo process) in steps S126 and S127 may be removed. Thatis, in the voice recognition apparatus 91, a change of a cluster may beallowed, and the undo process may not be performed.

The voice recognition apparatus 91 may be constructed such that whethersteps S126 and S127 should be performed or not can be selected (by auser or the like) from the outside of the voice recognition apparatus91.

In a case in which all members of a cluster are determined asunregistered words to be deleted in the unregistered word deletionprocess in FIG. 17, and thus all members are deleted, the unregisteredword deletion process is equivalent to deleting the cluster itself themembers belong to. In this case, it is not necessary to determine a newrepresentative member for that cluster (it is impossible to determine anew representative member) Therefore, in this case, after completion ofstep S122, steps S123 and S124 are skipped and the flow proceeds to stepS125 and then to S126. If it is determined in step S126 that no changein cluster structure has occurred, the process proceed to step S128. Instep S128, the maintenance module 140 updates the word dictionary storedin the dictionary buffer 134 on the basis of the score sheet updated(corrected) in step S122. Thereafter, the flow returns.

More specifically, in this case, all members of a certain cluster aredeleted, and thus the cluster itself is deleted. The maintenance module140 refers to the score sheet to detect the deleted cluster. Themaintenance module 140 then deletes an entry corresponding to thedeleted cluster from the word dictionary stored in the dictionary buffer134.

By deleting an entry corresponding to a certain cluster from the worddictionary stored in the dictionary buffer 134 in the above-describedmanner, amnesia or a loss in memory is realized.

Referring again to FIG. 16, if it is determined in step S101 that nounregistered word is commanded to be deleted, then, in step S102, thefeature extraction module 131 determines whether a voice has been input.

If it is determined in step S102 that no voice has been input, the flowreturns to step S101 to repeat the process from step S101.

That is, the feature vector deletion module 141 always checks whetherdeletion of an unregistered word (utterance information of anunregistered word stored in the feature vector buffer 137) has beencommanded, and the feature extraction module 131 always checks,independently of the feature vector deletion module 141, whether a voicehas been input.

Herein, if a user utters, then the uttered voice is passed through themicrophone and the AD converter in the input unit 107 thereby obtainingdigital voice data. The resultant digital voice data is supplied to thefeature extraction module 131.

In step S103, the feature extraction module 131 determines whether avoice has been input. If a voice is input (if it is determined that avoice has been input), then, in step S104, acoustic analysis isperformed on the voice data on a frame-by-frame basis thereby extractfeature vectors. The sequence of the extracted feature vectors issupplied to the matching module 132 and the unregistered word periodprocessing module 136.

Steps S104 to S108 are similar to steps S1 to S5 described above withreference to FIG. 9, and thus steps S104 to S108 are not describedherein.

As described above, if the feature vector deletion module 141 determinesthat the specific condition is satisfied, feature vector deletion module141 deletes, from the data stored in the feature vector buffer 137,utterance information (feature vector sequence in the example shown inFIG. 15) of a member regarded as having little influence on clusteringand associated data (ID, phoneme sequence, and storage time in theexample shown in FIG. 15), thereby suppressing the consumption of thestorage area without causing degradation in the function ofautomatically acquiring unregistered words.

Furthermore, the feature vector deletion module 141 corrects datarelated to the member (deletes unnecessary data) described in the scoresheet stored in the score sheet buffer 139 thereby further suppressingthe consumption of the storage area.

Furthermore, the maintenance module 140 updates the word dictionary onthe basis of the corrected score sheet. This makes it possible torealize amnesia or a loss in memory in the robot, thereby making therobot capable of providing greater entertainment to users.

In the embodiments described above, the steps described in the programstored in the storage medium may be performed either in time sequence inaccordance with the order described in the program or in a parallel orseparate fashion.

Furthermore, there are no particular limitations on the details of therespective modules shown in FIG. 14, as long as the modules providenecessary functions. For example, modules may be constructed by means ofhardware. In this case, a manufacturer may connect the respectivemodules in the manner shown in FIG. 14. In other words, instead of thevoice recognition unit 50A shown in FIG. 3, the apparatus realized bymeans of hardware constructed in the manner shown in FIG. 14 may be usedas the voice recognition unit.

In the embodiments described above, voice recognition is performed bymeans of the HMM method. However, in the present invention, voicerecognition may also be performed by means of another method such as aDP matching method. In the case in which voice recognition is performedby means of the DP matching method, the score employed in theabove-described embodiments correspond to the reciprocal of the distancebetween an input voice and a standard pattern.

In the embodiments described above, an unregistered word is clusteredand the unregistered word is registered in the word dictionary on thebasis of the result of clustering. The present invention may also beapplied to a word which is already registered in the word dictionary.

That is, because there is a possibility that different phoneme sequencesare obtained for utterances of the same word, if only one phonemesequence is registered for one word in the word dictionary, then thereoccurs a possibility that an utterance of the word is not correctlyrecognized as that word if the phoneme sequence obtained for theutterance is different from that registered in the word dictionary. Inthe present invention, the above problem can be solved as follows. Thatis, different utterances of the same word are clustered such thatacoustically similar utterances belong to the same cluster, and thedictionary is updated on the basis of the result of the clustering. As aresult, different phoneme sequences are registered for the same word inthe word dictionary, and thus it becomes possible to perform voicerecognition for various phonemes for the same word.

In the word dictionary, in addition to phoneme sequence, key informationmay be described in an entry corresponding to a cluster of anunregistered word, as described below.

For example, in the action decision unit 52, state recognitioninformation output from the image recognition unit 50B or the pressureprocessing unit 50C is supplied to the voice recognition unit 50A asrepresented by a broken line in FIG. 3 so that the maintenance unit 31(FIG. 4) in the voice recognition unit 50A receives the staterecognition information.

On the other hand, the feature vector buffer 28 and the score sheetmemory 30 also store an absolute time at which an unregistered word wasinput. On the basis of the absolute time described in the score sheetstored in the score sheet memory 30, the maintenance unit 31 detects thestate recognition information which was supplied from the actiondecision unit 52 when the unregistered word was input, and maintenanceunit 31 regards the detected state recognition information as keyinformation of that unregistered word.

The maintenance unit 31 registers, in the word dictionary, the staterecognition information as key information in an entry corresponding toa cluster of the unregistered word so that the entry includes the staterecognition information in addition to the phoneme sequence of therepresentative member of that cluster.

This makes it possible for the matching unit 23 to output, as a resultof voice recognition for the unregistered word registered in the worddictionary, the state recognition information registered as the keyinformation of the unregistered word. Furthermore, it becomes possibleto make the robot take an action in accordance with the staterecognition information registered as the key information.

More specifically, for example, when a word “red” is unregistered, ifthe CCD 16 detects an image of a red object, state recognitioninformation indicating that a red object has been detected is suppliedfrom the image recognition unit 50B to the voice recognition unit 50Avia the action decision unit 52. Herein, if a user utters “red”, whichis an unregistered word, then the voice recognition unit 50A determinesa phoneme sequence of the unregistered word “red”.

In this case, the voice recognition unit 50A adds, as an entry of theunregistered word “red” to the word dictionary, the phoneme sequence ofthe unregistered word “red” and the key information indicating the staterecognition information “red”.

If a user utters “red” thereafter, the phoneme sequence of theunregistered word “red” registered in the word dictionary has a highscore for the utterance, and thus the voice recognition unit 50Aoutputs, as a result of voice recognition, the state recognitioninformation “red” registered as the key information.

The result of the voice recognition is supplied from the voicerecognition unit 50A to the action decision unit 52. In response, forexample, the action decision unit 52 may make the robot search theenvironment for the red object on the basis of the output from the imagerecognition unit 50B and walk toward the red object.

That is, in this specific example, although the robot cannot recognizean utterance of “red” when the robot encounters it for the first time,if a user utters “red” when the robot is detecting an image of an redobject, then the robot relates the utterance “red” to the red objectbeing detected as the image, thereby making is possible for the robotto, when the user utters “red” thereafter, recognize the utterancecorrectly as “red” and walk toward a red object present in theenvironment. This gives the user an impression that the robot grows bylearning what the user speaks.

The voice recognition apparatus 91 shown in FIG. 13 may also operate ina similar manner.

Although in the embodiments described above, the scores are stored inthe score sheet, the scores may be recalculated as required.

Although in the embodiments described above, the detected cluster isdivided into two clusters, the detected cluster may be divided intothree or more clusters. The detected cluster may also be divided into anarbitrary number of clusters such that the distances among the clustersbecome greater than a predetermined value.

In the embodiments described above, not only scores but also phonemesequences, cluster numbers, and representative member IDs are registeredin the score sheet (FIG. 8). Information other than the scores is notnecessarily needed to be registered in the score sheet, but may bestored and managed separately from the scores.

INDUSTRIAL APPLICABILITY

According to the present invention, a cluster to which an input voice isto be added as a member is detected from existing clusters obtained byclustering voices. The input voice is added as a new member to thedetected cluster, and the cluster is divided depending on the members ofthe cluster. On the basis of the result of the division, the dictionaryis updated. Thus, it becomes possible to easily register, into thedictionary, a word which has not been registered in the dictionarywithout resulting in a significant increase in the size of thedictionary.

1. A voice recognition apparatus for processing an input voice andupdating a dictionary used in a language processor in accordance with aresult of the processing of the input voice, said voice recognitionapparatus comprising: cluster detection means for detecting, fromexisting clusters obtained by clustering voices, a cluster to which saidinput voice is to be added as a new member; cluster division means foremploying said input voice as the new member of the cluster detected bysaid cluster detection means and dividing said cluster depending onmembers of said cluster; and update means for updating the dictionary onthe basis of a result of division performed by said cluster divisionmeans wherein the dictionary stores a phoneme sequence of a vocabularyto be recognized, and wherein the update means updates the dictionary byadding, as a new entry to the dictionary, a phoneme sequence of a voicecorresponding to a representative member representing members of acluster created by the division or by replacing an entry of thedictionary with the phoneme sequence of the voice corresponding to therepresentative member representing members of the cluster created by thedivision.
 2. A voice recognition apparatus according to claim 1, whereinsaid cluster division means divides the cluster by means of an EM(Expectation Maximum) method.
 3. A voice recognition apparatus accordingto claim 1, wherein said cluster detection means calculates the score ofthe input voice with respect to each member of the cluster bydetermining the likelihood that the input voice is observed in themember of the cluster; and selects, from the members of the cluster, amember giving a highest value to said score of the input voice andemploys the selected member as a representative member representing themembers of the cluster; and determines the cluster having saidrepresentative member as a cluster to which the said input voice it tobe added as a new member.
 4. A voice recognition apparatus according toclaim 1, wherein said input voice is an unregistered word which has notbeen registered, in advance, in the dictionary.
 5. A voice recognitionapparatus according to claim 3, wherein in a case in which, of membersof the cluster, a member having a greatest sum of scores with respect tothe other members of that cluster is employed as the representativemember representing the members of that cluster, said cluster divisionmeans divides the cluster, to which the input voice has been added, intotwo clusters, that is, first and second clusters, such that two membersof the original cluster become representative members of the first andsecond clusters, respectively.
 6. A voice recognition apparatusaccording to claim 5, wherein in a case in which there are a pluralityof combinations of two clusters consisting of first and second clusters,said cluster division means divides a cluster including the input voiceas a member thereof into two clusters such that the cluster-to-clusterdistance between the first cluster and the second cluster becomessmallest.
 7. A voice recognition apparatus according to claim 6, whereinwhen a combination of two clusters consisting of first and secondclusters is selected such that the cluster-to-cluster distance betweenthe first cluster and the second cluster becomes smallest, if saidsmallest cluster-to-cluster distance is greater than a predeterminedthreshold, then said cluster division means divides the clusterincluding the input voice as a member thereof into said two clusters. 8.A voice recognition apparatus according to claim 5, further comprisingstorage means for storing the scores of the members of said cluster withrespect to each member of each score.
 9. A voice recognition apparatusaccording to claim 1, wherein the dictionary stores a phoneme sequenceof a vocabulary to be recognized, and wherein said voice recognitionapparatus further comprises voice recognition means for recognizing avoice on the basis of an acoustic model constructed in accordance withthe phoneme sequence stored in said dictionary.
 10. A voice recognitionapparatus according to claim 9, wherein said acoustic model is an HMM(Hidden Markov Model).
 11. A voice recognition apparatus according toclaim 9, wherein said voice recognition means constructs an acousticmodel corresponding to a phoneme sequence stored in the dictionary byconcatenating HMMs in units of sub-words and recognizes a voice on thebasis of said acoustic model.
 12. A voice recognition apparatusaccording to claim 9, wherein said voice recognition means recognizes avoice also on the basis of a predetermined grammatical rule.
 13. A voicerecognition apparatus according to claim 12, wherein said voicerecognition means extracts a particular period of the input voice inaccordance with the predetermined grammatical rule; and said clusterdetection means and said cluster division means perform their processeson said period of the input voice.
 14. A voice recognition apparatusaccording to claim 13, wherein said voice recognition means extracts, assaid particular period, a period of an unregistered word which is notregistered in the dictionary from the input voice.
 15. A voicerecognition apparatus according to claim 14, wherein said voicerecognition means extracts the period of the unregistered word inaccordance with the predetermined grammatical rule using a garbagemodel.
 16. A voice recognition apparatus according to claim 1, furthercomprising: storage means for storing voice information associated withthe input voice for use by the cluster detection means to detect acluster; and deletion means for, when it is determined that a specificcondition is satisfied, deleting a particular one of pieces of voiceinformation stored in the storage means.
 17. A voice recognitionapparatus according to claim 16, wherein the voice information stored inthe storage means is digital data of the input voice.
 18. A voicerecognition apparatus according to claim 17, further comprising featureextraction means for extracting a feature vector indicating a specificfeature of the input voice from the digital data of the input voice,wherein the voice information stored in the storage means is the featurevector of the input voice extracted by the feature extraction means. 19.A voice recognition apparatus according to claim 16, wherein saiddeletion means determines that the specific condition is satisfied whenthe number of members belonging to the specific cluster is greater thana predetermined value.
 20. A voice recognition apparatus according toclaim 16, further comprising not-referred-to time calculation means forcalculating the not-referred-to time during which the cluster has notbeen referred to, wherein said deletion means determines that thespecific condition is satisfied when the not-referred-to time of thecluster calculated by the not-referred-to time calculation means isgreater than a predetermined value.
 21. A voice recognition apparatusaccording to claim 16, further comprising input means for inputting atrigger signal, wherein said deletion means determines that the specificcondition is satisfied when the trigger signal is input via the inputmeans.
 22. A voice recognition apparatus according to claim 16, furthercomprising emotion control means for controlling a parameter associatedwith emotion, wherein said deletion means determines that the specificcondition is satisfied when the value of the parameter associated withemotion controlled by the emotion control means is greater than apredetermined value.
 23. A voice recognition apparatus according toclaim 16, further comprising used-memory-area calculation means forcalculating the amount of used memory area of the storage means, whereinsaid deletion means determines that the specific condition is satisfiedwhen the amount of used memory area calculated by the used-memory-areacalculation means is greater than a predetermined value.
 24. A voicerecognition apparatus according to claim 16, further comprisingclustering means for reclustering a voice corresponding to voiceinformation stored in the storage means.
 25. A voice recognitionapparatus according to claim 24, wherein said update means updates thedictionary also on the basis of a result of reclustering performed bythe clustering means.
 26. A voice recognition apparatus according toclaim 24, further comprising representative member selection means for,when the voice information is deleted by the deletion means, selecting arepresentative member representing the members of the cluster to whichthe voice corresponding to the deleted voice information belongs beforebeing deleted, wherein if the new representative member selected by therepresentative member selection means is different from a previousrepresentative member, said clustering means reclusters all voiceinformation stored in the storage means.
 27. A voice recognitionapparatus according to claim 26, further comprising deletion cancelmeans for if the structure of the cluster reclustered by the clusteringmeans is different from the immediately previous structure the clusterhad before being reclustered by the clustering means, canceling thedeletion process performed, by the deletion means, on the voiceinformation such that the original state is obtained.
 28. A voicerecognition apparatus according to claim 26, wherein said clusteringmeans performs reclustering by means of a k-means method.
 29. A voicerecognition method for processing an input voice and updating adictionary used in a language processor in accordance with a result ofthe processing of the input voice, said voice recognition methodcomprising the steps of: detecting, from existing clusters obtained byclustering voices, a cluster to which said input voice is to be added asa new member; employing said input voice as the new member of thecluster detected in said cluster detection step and dividing saidcluster depending on members of said cluster; and updating thedictionary on the basis of a result of division performed in saidcluster division step wherein the dictionary stores a phoneme sequenceof a vocabulary to be recognized, and wherein the update means updatesthe dictionary by adding, as a new entry to the dictionary, a phonemesequence of a voice corresponding to a representative memberrepresenting members of a cluster created by the division or byreplacing an entry of the dictionary with the phoneme sequence of thevoice corresponding to the representative member representing members ofthe cluster created by the division.
 30. A computer-readable mediumstoring a computer program for causing a computer to perform voiceprocessing for processing an input voice and updating a dictionary usedin a language processor in accordance with a result of the processing ofthe input voice, said program comprising the steps of: detecting, fromexisting clusters obtained by clustering voices, a cluster to which saidinput voice is to be added as a new member; employing said input voiceas the new member of the cluster detected in said cluster detection stepand dividing said cluster depending on members of said cluster; andupdating the dictionary on the basis of a result of division performedin said cluster division step wherein the dictionary stores a phonemesequence of a vocabulary to be recognized, and wherein the update meansupdates the dictionary by adding, as a new entry to the dictionary, aphoneme sequence of a voice corresponding to a representative memberrepresenting members of a cluster created by the division or byreplacing an entry of the dictionary with the phoneme sequence of thevoice corresponding to the representative member representing members ofthe cluster created by the division.